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ABSTRACT 



This letter reviews actions by the New York State Education 
Department and New York City Board of Education as of January 2002 to 
implement recommendations contained in a prior audit report. The audit found 
that the Department and the Board did not always perform adequate analysis to 
determine the effectiveness of various consultant programs being used by 
schools to improve performance on state standardized tests. It also found 
that improvement programs were not always effective, and some schools were 
able to improve performance without any program help. The follow-up review 
found that the Department and the Board made considerable progress in 
implementing prior audit report recommendations. This letter describes 
improvements related to the six prior audit report recommendations : 
developing the capability to adequately determine which resource allocation 
and program initiatives work most and least effectively at improving academic 
performance; ensuring that all school improvement programs are evaluated 
independently for effectiveness; determining why some school improvement 
programs are more effective than others; monitoring performance trends at 
below- standard schools and determining which schools are improving and why; 
ensuring that any revised performance data submitted by schools are verified 
or appropriately documented; and developing a method for evaluating the 
performance of schools with small test populations. (SM) 
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from the original document. 







^r 
\o 
' ^ ■ 



Q 



W 



Office of the State Comptroller, State of New York [Letter to the 
Commissioner of Education and the Chancellor of New York City Board 
of Education: Follow-up Review of Monitoring Improvement Efforts of 
Schools with Low Standardized Test Scores ] 

March 8, 2002 

Jerry Barber 

Office of the State Comptroller 



U S DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

k / This document has been reproduced as 
v received from the person or organization 
originating it. 

□ Minor changes have been made to 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



twi- 

ce 

On 

CO 

o 




best COPT 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



2 






RESOURCES 

INFORMATION CENTER (ERIC) 



1 



H. CARL McCall 
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STATE OF NEW YORK 

OFFICE OF THE STATE COMPTROLLER 

March 8, 2002 



UT) 



Mr. Richard Mills 

Commissioner of Education 

New York State Education Department 

State Education Building 

Albany, New York 12234 

Mr. Harold O. Levy 
Chancellor 

New York City Board of Education 
110 Livingston Street 
Brooklyn, NY 1 1201 



Re: Report 2001-F-53 



Dear Mr. Mills and Mr. Levy: 

Pursuant to the State Comptroller’s authority as set forth in Article V, Section 1 of the State 
Constitution, Article II, Section 8 of the State Finance Law, and Article ID, Section 33 of the General 
Municipal Law, we have reviewed the actions taken by the New York State Education Department 
(Department) and New York City Board of Education (Board) as of January 3 1 , 2002 to implement 
the recommendations contained in our audit report, Monitoring Improvement Efforts of Schools With 
Low Standardized Test Scores (Report 98-S-34). Our report, which was issued on May 10, 2000, 
assessed the actions taken by the Department and the Board in monitoring the improvement efforts 
of Schools Under Registration Review (SURR) and other schools performing below the State 
standard. 

Background 



The Board of Regents (Regents) is responsible for setting education policies and for guiding, 
managing and monitoring the education system in New York State. In fulfilling these 
responsibilities, the Regents work with the Governor and Legislature, who also initiate education 
programs and ultimately control the State funds devoted to education. The 16 Regents are elected for 
five-year terms by the Legislature and are headed by a Chancellor. The Regents are served by the 
Department, which administers the State’s education policies and programs. The Department 
oversees local school boards and monitors compliance with education laws and regulations. 
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The State education system is primarily a decentralized operation, rooted in the concept of 
local autonomy. There are about 700 local school boards across the State making daily operating 
decisions for about 4,111 individual schools. Further, the Board oversees all public schools in 
New York City. The Regents also supervises the Board, but various sections of the State 
Education Law assign many responsibilities specifically to the Board. The Regents require that all 
students take certain standardized tests to measure whether they have achieved the State learning 
standards. As a result, each year the Department analyzes how students at each school in the State 
performed on these tests. Based on the criteria established in the Commissioner’s Regulations, 
each year the Commissioner determines which schools achieved State standards, were below 
standards, or were farthest from State standards. Schools that are farthest from State standards, 
and are determined by the Commissioner to be most in need of improvement, are identified as 
Schools Under Registration Review (SURR). Each school in this category is visited regularly and 
monitored by Department staff that review the school’s operations and recommend operational 
improvements. 

Summary Conclusions 

In our prior audit, we found that the Department and the Board did not always perform 
adequate analysis to determine the effectiveness of various consultant programs being used by 
schools to improve performance on State standardized tests. Also, we found that the improvement 
programs were not always effective and some schools were able to improve their performance 
without the help of these costly programs. 

In our follow-up review, we found that the Department and the Board have made 
considerable progress in implementing our prior audit report recommendations. For example, the 
Department is using the school “report cards” to monitor trends at low performing schools and is 
also developing a survey instrument to help identify programs which work most effectively. The 
Department and the Board have also taken steps to strengthen efforts to evaluate improvement 
programs. 

Summary of Status of Prior Audit Recommendations 

Of the six prior audit report recommendations, four were applicable to the Department and 
two were applicable to the Department and the Board. The Department implemented three and 
partially implemented one of the four recommendations applicable to it. The two recommendations 
applicable to the Board and the Department were partially implemented based primarily on steps 
undertaken by the Department. 

Follow-up Observations 

To the Department: 

Recommendation 1 

Develop the capability to adequately determine which resource allocation, and which program 
initiatives work most effectively and which work less effectively at improving academic performance 
in given school settings. 
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Status - Partially Implemented 

Agency Action - The Department, in collaboration with its New York City Regional Network at the 
Teacher Center (Center), and a consultant, the Center for Resource Management (CRM), is 
developing a survey instrument to help Department officials determine what they should do 
to improve student academic achievement. This includes the identification of specific 
programs to improve academic performance. However, at the time of our follow-up review, 
work on the instrument had not been completed. The Center was field-testing the 
instrument to ensure that it met the Department’s standards for validity and reliability. 
Although the Center did develop a timeline for the completion of the project, due to 
circumstances beyond its control (e.g. the September 11, 2001 incident), the completion is 
delayed. Department officials anticipate completion of this project by September 2002. 

To the Department and the Board: 

Recommendation 2 

Ensure that all school improvement programs are evaluated independently to determine the extent to 

which they are effective at improving student scores on standardized tests. 

Status - Partially Implemented 

Agency Action - Department officials indicated that they have authority to evaluate only those 
programs directly under Department control. Nonetheless, the Department and the Board 
are participating in the statewide Comprehensive School Reform Demonstration (CSRD) 
program, which is a federal initiative that is designed to increase student achievement 
through the use of comprehensive research-based reform models. The CSRD program is 
designed for completion over a period of three years ending in the fall of 2002. The CSRD 
includes focus group interviews, onsite visits, case studies, and a survey to identify 
programs that work effectively. 

The Department has also worked with the Center and the CRM to develop several 
technical manuals and guides that help district officials to choose a CSRD model that offers 
the "best fit" to a school’s program of services. These manuals also address the correlation 
between the models and State guidance on teaching and assessment. According to 
Department officials, these documents are made available to the districts through the 
Department website, workshops, training, the CSRD newsletter and manuals/guides for staff 
use. 

The Board’s Division of Assessment and Accountability also evaluated selected 
CSRD reform programs. However, this evaluation generally did not address the extent to 
which individual school improvement programs are effective. The Board’s Comprehensive 
Evaluation Report concluded that a definite statement regarding the success of certain 
programs could not be made, because of the small number of schools utilizing research- 
based programs. In addition, the Board stated that it is over simplistic to attribute 
improvement in student achievement simply to the impact of a single program. 
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Furthermore, Board officials suggested that in their view certain other factors must be 
considered in evaluating programs’ effectiveness. The Board stated that similar to the CSRD 
program, the programs adopted by schools are already research-based, that many programs 
require students to acquire skills before achievement can be raised, and that even sound 
researched programs are influenced by other factors that contribute to the success or failure 
of the initiative. 

To the Department and the Board: 



Recommendation 3 



Determine why some school improvement programs are more effective than others, and determine 
whether certain types of programs are likely to be more effective in certain circumstances. 

Status - Partially Implemented 

Agency Action - The Westchester Institute for Human Services Research, Inc. (a Department 
consultant) and the CRM formally studied the CSRD initiative. The study indicated that 
CSRD schools implemented reform models in very different ways. The study also 
suggested that some school reform models might be more successful than others in 
broadening teacher use of good instructional practices and improving educational outcomes. 
The study rates the overall assessment of the reform models as encouraging. In addition, 
officials developed a series of CSRD technical manuals to assist school districts to select the 
kind of programs that work best for their environment and is presently working on phase 
three of the technical manual aimed at improving academic performance. 

The Board’s Division of Assessment and Accountability also assessed the CSRD 
program through a survey that asked teachers and district CSRD liaisons about changes in 
program models. A report issued during the 1999-2000 year indicated that teachers had 
changed teaching methods as a result of the reform effort and parents received increased 
information regarding the new learning standards. Also, the report indicated that the school 
reform model significantly enhanced curriculums and instructional materials. 

The Board, however, questioned the benefit of evaluating individual improvement 
programs after New York City schools implemented them. Board officials advised us that 
programs adopted by schools are research-based and have previously contributed to 
increased academic achievement outside of New York City. Officials also stated that there 
are limitations in determining why some school programs are more effective than others. 
Officials noted that implementing programs in certain schools might take additional time 
due to organization changes and school culture. Further, the Board stated that systematic 
reform does not happen quickly. Rather, it often takes several years. 




6 



To the Department: 



Recommendation 4 



Monitor performance trends at below-standard schools, determine which schools are improving, 
identify the reasons for the improvement, and recommend that the effective practices be adopted at 
the below-standard schools where performance is not improving. 

Status - Implemented 

Agency Action - Each year, the Department publishes a “report card” for each public school in the 
State. The report card notes a school’s performance on tests for English language arts and 
mathematics in the fourth and eighth grades. The Department also issues report cards for 
high schools based on students’ performance on Regents examinations. The report cards 
indicate performance over the preceding three-year period, and thereby, provide an indication 
of trends in schools’ performance on the standardized tests. 

In addition, Department officials evaluated 100 schools that have improved 
performance on the elementary English language arts assessment for 1999, 2000 and 2001. 
The evaluation indicated areas where improvement was strong and recommended that local 
school boards implement similar strategies through the System of Accountability for Student 
Success (S ASS) initiative. Officials further indicated that low performing schools can access 
the Department’s “Sharing Success” internet site to research programs that other districts 
consider successful. 

To the Department: 



Recommendation 5 



Ensure that any revised performance data submitted by a school is either verified or appropriately 
documented. 

Status - Implemented 

Agency Action - Department officials stated that presently there are four steps being used to verify 
student data. These steps include internal data validation, actual sign-off by superintendents, 
continuous Department checks including audits; and checks performed by the Regional 
Information Centers which process student data for school districts outside of the big five 
cities. Furthermore, the Department is providing technical assistance to school districts to 
assist in the completion of the data forms. 

In addition, the Department is continuing efforts to improve performance data 
through development of the individual student data collection system with student 
identification numbers that are unique statewide. In the past, the Department has relied on 
aggregated school-level data to monitor school’s performance. Beginning with the 2001- 
2002 school year, the Department will collect all student performance data in individual 



student records. The records will be submitted with a student identification number unique 
to the school district. These efforts should minimize the need to revise student performance 
data. 

To the Department: 



Develop a method for evaluating the performance of schools with small test populations. 

Status - Implemented 

Agency Action - According to Department officials, this has been accomplished through SASS. 
Specifically, by using test data from multiple years for a school with fewer than 20 students, 
officials develop a performance index that is compared to the State standard for a given year. 
The long-term goal is that 90 percent of students achieve the standard. With regard to the 
standards, schools are placed in one of the three categories such as meeting the State 
standard, below standard, and farthest from standard. Preliminary results indicated that 17 
schools with fewer than 20 students were the farthest from the standard category. However, 
most all of these schools serve special education students only, which explains why their 
performance was low. This evaluation also disclosed that the small number of rural schools 
with small test populations generally performed adequately on the tests. 

Major contributors to this report were Brian Mason, Arthur F. Smith, Don Wilson and 
Alexander Marshall. 

We would appreciate your response to this report within 30 days, indicating any actions 
planned to address the unresolved issues discussed in this report. We also thank Department and 
Board management and staff for the courtesies and cooperation extended to our auditor during this 
review. 
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Very truly yours 




Jerry Barber 
Audit Director 



cc: 



Deirdre Taylor, DOB 
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